What does the crowd believe? A hierarchical approach to estimating subjective beliefs from empirical data

People’s beliefs about everyday events are both of theoretical interest in their own right and an important ingredient in model building—especially in Bayesian cognitive models of phenomena such as logical reasoning, future predictions, and language use. Here, we explore several recently used methods for measuring subjective beliefs about unidimensional contiguous properties, such as the likely price of a new watch. As a first step towards a way of assessing and comparing belief elicitation methods, we use hierarchical Bayesian modeling for inferring likely population-level beliefs as the central tendency of participants’ individual-level beliefs. Three different dependent measures are considered: (i) slider ratings of (relative) likelihood of intervals of values, (ii) a give-a-number task, and (iii) choice of the more likely of two intervals of values. Our results suggest that using averaged normalized slider ratings for binned quantities is a practical and fairly good approximator of inferred population-level beliefs.